Proximal Iteratively Reweighted Algorithm with Multiple Splitting for Nonconvex Sparsity Optimization

This paper proposes the Proximal Iteratively REweighted (PIRE) algorithm for solving a general problem, which involves a large body of nonconvex sparse and structured sparse related problems. Comparing with previous iterative solvers for nonconvex sparse problem, PIRE is much more general and efficient. The computational cost of PIRE in each iteration is usually as low as the state-of-the-art convex solvers. We further propose the PIRE algorithm with Parallel Splitting (PIRE-PS) and PIRE algorithm with Alternative Updating (PIRE-AU) to handle the multi-variable problems. In theory, we prove that our proposed methods converge and any limit solution is a stationary point. Extensive experiments on both synthesis and real data sets demonstrate that our methods achieve comparative learning performance, but are much more efficient, by comparing with previous nonconvex solvers. Introduction This paper aims to solve the following general problem min x∈Rn F (x) = λf(g(x)) + h(x), (1) where λ > 0 is a parameter, and the functions in the above formulation satisfy the following conditions: C1 f(y) is nonnegative, concave and increasing. C2 g(x) : R → R is a nonnegative multi-dimensional function, such that the following problem min x∈Rn λ〈w,g(x)〉+ 1 2 ||x− b||2, (2) is convex and can be cheaply solved for any given nonnegative w ∈ R. C3 h(x) is a smooth function of type C, i.e., continuously differentiable with the Lipschitz continuous gradient ||∇h(x)−∇h(y)|| ≤ L(h)||x−y|| for any x,y ∈ R, (3) L(h) > 0 is called the Lipschitz constant of∇h. C4 λf(g(x)) + h(x)→∞ iff ||x||2 →∞. ∗Corresponding author. Copyright © 2014, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. Note that problem (1) can be convex or nonconvex. Though f(y) is concave, f(g(x)) can be convex w.r.t x. Also f(y) and g(x) are not necessarily smooth, and h(x) is not necessarily convex. Based on different choices of f , g, and h, the general problem (1) involves many sparse representation models, which have many important applications in machine learning and computer vision (Wright et al. 2009; Beck and Teboulle 2009; Jacob, Obozinski, and Vert 2009; Gong, Ye, and Zhang 2012b). For the choice of h, the least square and logistic loss functions are two most widely used ones which satisfy (C3): h(x) = 1 2 ||Ax−b||2, or 1